1. Field of the Invention
This invention relates to an image data decoding and displaying method and an apparatus therefor. In this method and apparatus, decoded frame data are sequentially stored in a memory and read therefrom to display. This invention can be applied, for example, to an MPEG decoder.
2. Description of the Prior Art
Improving technology for compressing images into codes and storing them into various record media such as CD-ROM or DAT has become popular, as represented by the international encoding standards MPEG. Today home appliance manufacturers and computer manufacturers are making efforts to develop multimedia information home appliances and intend to market goods which meet MPEG. Below is an outline of processing based on MPEG.
FIG. 1 is a typical diagram showing the flow of image encoding and decoding based on MPEG. As shown in FIG. 1, images input from an image input device 2, such as a camera, are compressed and encoded by a video encoder 4. Quantization processes and DCT (discrete cosine transform) are usually involved in encoding. Encoded data is written into a record medium 6.
Decoding follows a reverse process: data is read from the record medium 6 and decoded by a video decoder 8. Decoding is preceded by a reverse quantization process and reverse DCT. Image data decoded by the video decoder 8 is output in a displayable format at a displayable timing and reproduced and displayed by a display device 10. Decoding is performed with forward prediction from past reproduced images and reverse prediction from future reproduced images (bidirectional prediction), as necessary.
FIG. 2 is a schematic diagram of a GOP (group of pictures) of MPEG. Fifteen screens in FIG. 2 (called pictures in MPEG) can be regarded to be taken in this order. These fifteen pictures, called a GOP in MPEG, form one predictive process unit. That is, encoding and decoding can be done through cross reference to pictures within one GOP, wherein a GOP is a unit of random access. Motion images can be encoded and decoded by continuously processing sets, each of which is comprised of one GOP and its sequence header.
As shown in FIG. 2, there are three kinds of pictures: I, P, and B. An I picture (image encoded within its frame) is limitedly encoded within its own frame and reference to other pictures is not necessary for decoding. A P picture (image encoded with forward prediction) requires forward prediction; only past reproduced images are necessary for decoding. However, a B picture (image encoded with bidirectional prediction) requires bidirectional prediction; I and P pictures which are reproduced after the B picture will also be referenced. In FIG. 2 the direction of prediction is shown with an arrow. When a GOP is actually encoded, the data of I or P pictures input after the B picture must be known. Therefore, picture I2, for example, is encoded before pictures B0 and B1 and the encoded data is written into the record medium 6. In FIG. 2, the period M of I or P pictures applied to bidirectional prediction is three.
FIG. 3 is a diagram showing the decoding and reproducing order based on MPEG. The upper row of FIG. 3 indicates the decoding order, while the lower row indicates the reproducing order. The decoding order is the same as the encoding order; that is, the decoding order corresponds to the order in which pictures are arranged in the record medium 6. Therefore, on the decoding side, I2, B3, B4, . . . pictures are read in this order from the record medium 6, decoded, re-ordered, and then output in the original order. In FIG. 3, the I2 picture must not be output before the B0 and B1 pictures, so the I2 picture is held in an internal memory until the B0 and B1 pictures are output. Similarly, other I and P pictures must be held in a memory until their corresponding B pictures are output. As shown in FIG. 3, decoded B pictures are output as soon as possible in order to enable adoption of a small capacity memory.
FIG. 4 is a diagram showing the process timing of decoding and display by a conventional video decoder 8. The lower part of FIG. 4 shows memory banks in the video decoder 8 and pictures held in each bank at each timing. This memory has 4 banks for reasons described below. Each bank corresponds to an area in which one frame (one picture) of data is stored.
As shown in FIG. 4, periods of frames FL1-6 depend on vertical synchronizing signals. Each frame period consists of two field periods. It is assumed that the low state of a field signal indicates the first field period and the high state of the field signal indicates the second field period.
In FIG. 4, the time at which I2, B0, and B1 pictures of some GOP have already been decoded is the starting point of FL1. Subsequently, P5, B3, B4, . . . pictures will be decoded as in the decoding order of FIG. 3. However, with display operation, the starting point of FL1 occurs only when picture B0 has been displayed. Subsequently, the B1, I2, . . . pictures are displayed as in the reproducing order of FIG. 3. Decode and display operation in each frame period will be described below.
(1) FL1
The B1 picture is displayed. To do this, the B1 picture is held in a bank (assumed to be a bank 1) until the end of FL1. The I2 picture to be displayed in FL2 is still held in another bank (assumed to be bank 0). Apart from display operation, the P5 picture is decoded. To store the decoded data, still another bank (assumed to be a bank 2) is assigned. Decoding is done in macroblocks (one macroblock is 16 pixels by 16 lines) and data is stored in a memory in macroblocks. In this frame a bank 3 is an empty area (unused).
(2) FL2
The I2 picture is displayed. Picture I2 is held in bank 0 until the end of this frame. Picture P5 to be displayed later is still held in the bank 2. The data of picture B1 which has already been displayed is unnecessary; therefore, bank 1 is assigned to picture B3 to be decoded. In this frame, too, bank 3 is an empty area.
(3) FL3
The I2 picture had been displayed, but it is still held in bank 0 because the reference to picture I2 is needed for picture B4 which is being decoded. Picture B3 which is being displayed and picture P5 to be displayed later stay in the banks. Picture B4 which is being decoded is stored in macroblocks in bank 3. To achieve this, the memory must have four banks.
In this way a timing chart of FIG. 4 is obtained according to the rules: "a B picture which has been already displayed is unnecessary" and "I and P pictures are held for a certain period to decode the B pictures between them." As shown in FIG. 4, all four banks are used every three frame periods (FL3, 6, 9, . . . ).
As was stated, the conventional video decoder 8 requires a memory having four banks to hold a decoded picture until the subsequent B pictures are decoded and to hold the present picture until it is actually displayed.
Memory capacity necessary to reserve four banks will be described below. For a screen of an NTSC system having 352 pixels.times.240 lines, one frame corresponds to about 123.8 Kbytes of data, and four frames will have 495.2 Kbytes of data. In addition, a 40-50 Kbyte temporary buffer called a VBV buffer should be prepared with MPEG, resulting in a total capacity of about 540-550 Kbytes. This exceeds 4 Mbits, and so one 4 Mbit DRAM will result in a shortage of capacity. For a screen of a PAL system having 352 pixels.times.288 lines, one frame corresponds to about 148.5 Kbytes of data. Thus, the same problem occurs.
For the above reason, the conventional video decoder 8 increases its memory capacity, for example, by adding one 1 Mbit DRAM to one 4 Mbit DRAM. To make an apparatus smaller and less expensive, it is desirable to use only one 4 Mbit DRAM. To solve this problem, the following method, for example, based on the above prior art is possible. A B picture which cannot be stored in a memory is discarded every three frames and the picture displayed in the preceding frame is simply displayed twice. However, such relaxation of specifications will result in lowered image quality, and is therefore not desirable.